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Abstract 

This  project  was  intended  to  solve  the  problem  of  the  lack  of 
physician  knowledge  at  Irwin  Army  Community  Hospital  about  the 
Workload  Management  System  for  Nursing  (WMSN).  It  explores  what 
nurses  think  physicians  should  know  about  nursing  to  perform  their 
physician  tasks  in  a  manner  that  uses  the  scarce  resources  of  nursing 
in  the  most  efficient  way  possible.  The  objectives  of  the  study  were 
to:  determine  what  the  physician  needs  to  know,  and  create  a  method 
for  teaching  it;  and  evaluate  the  change  in  physician  knowledge  that  is 
expected  to  result  from  the  education.  An  interview  process  was  used 
with  a  group  of  nurses  familiar  with  the  WMSN  to  determine  what 
physicians  needed  to  know  about  the  WMSN.  This  resulted  in  the 
development  of  a  survey  instrument  that  represents  the  sets  of 
WMSN  knowledge  required  by  physicians.  The  second  step  of  the 
project  was  to  develop  a  curriculum  based  on  the  knowledge  sets. 

This  initial  curriculum  was  taught  to  a  group  of  physicians  at  Irwin 
Army  Community  Hospital.  The  physicians  took  a  pre-test  and  a  post- 
test,  to  assist  in  validating  the  change  in  their  knowledge  and  to 
adjust  the  material  to  their  needs.  Two  iterations  of  curriculum 
development  occurred.  A  final  curriculum  was  developed  based  on 
lessons  learned  dining  the  teaching  phase. 
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CHAPTER  I 
Introduction 

Conditions  Which  Prompted  the  Study 

Scarce  resources  have  always  determined  the  methods  by  which 
management  solves  problems  and  provides  services.  Nurses  have 
become  a  scarce  resource.  Dollars  have  become  even  more  scarce. 

The  scarcity  of  nursing  and  fiscal  resources  is  driving  a  review  of 
utilization  methods.  In  the  past,  muses  simply  complied  with  a 
physician’s  orders,  even  if  this  required  a  higher  level  of  staffing  for 
that  area  or  ward.  Nurses  have  long  felt  that  physicians  are  often  not 
aware  that  their  orders  may  not  always  reflect  efficient  use  of  the 
nursing  staff  and  that  at  times,  those  orders  may,  in  fact,  do  the 
opposite  by  creating  unnecessary  workload  and  expense  in  the  nursing 
arena.  Efficient  use  of  future  resources  demands  that  physicians 
know  the  effect  of  their  orders  on  the  resources  of  the  nursing 
department. 

Statement  of  the  Management  Problem 

Physicians  at  Irwin  Army  Community  Hospital  do  not  know 
enough  about  the  Workload  Management  System  for  Nurses  to  use  it 
effectively.  The  goal  of  this  project  was  to  create  a  curriculum  for 
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physicians  that  has  a  built-in  evaluation  system  for  changing  the 
curriculum  as  physician  needs  change. 

Review  of  the  Literature 

Overview.  Prospective  payment  has  created  a  desire  for 
improved  internal  operating  efficiency  by  nurse  executives  and 
hospital  administrators.  A  shortage  of  nurses  who  are  active  in  the 
practice  of  nursing  has  also  added  to  the  need  for  internal  operating 
efficiency,  especially  in  the  inpatient  setting.  Since  nursing  managers 
are  responsible  for  maintaining  the  qualify  of  nursing  care,  they  must 
have  a  systematic  means  for  assessing  the  proper  distribution  of  their 
staff  resources  in  proportion  to  the  needs  of  the  patients  and 
considering  their  current  expenditures  and  for  projecting  future  needs. 
A  well-established  patient  acuity  system  (sometimes  called  a  patient 
classification  system )  tailored  to  a  given  institution’s  resources, 
standing  operating  procedures,  and  working  environment,  is  a 
powerful  tool  for  controlling  operations  and  justifying  managerial 
decisions. 

In  the  dictionary  sense,  classification  is  a  means  of  grouping, 
ordering,  or  arranging  objects  or  concepts  into  sets,  based  on 
relationships  among  the  objects  or  concepts.  The  term  inpatient 
classification  is  frequently  used  within  the  nursing  profession  and  has 
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come  to  refer  specifically  "to  the  hundreds  of  different  methods  and 
procedures  used  to  group  patients  according  to  their 
requirements  for  nursing  care  for  the  purposes  of  nursing  staff 
determination  and  allocation"  (Giovannetti,  1985,  p.  88). 

In  the  nursing  field,  an  acuity  system  is  "a  measurement  tool 
that  provides  data  to  assist  nurse  executives  in  justifying  the  number 
of  RNs  and  ;»araprofessionals  necessary  to  staff  an  acute  care  unit" 
(Adv?.np  &  Johnson,  1986,  p.  21).  A  patient  classification  or  patient 
acmty  ;  stem  assumes  "that  the  nursing  process  is  measurable,  that 
nursing  process  activities  constitute  a  sat  of  interrelated  functions 
which  are  measurable,  and  that  the  needs  of  the  patients  can  be 
identified"  (Philibert,  1986,  p.  60).  Through  physician  orders  and 
nursing  assessment,  each  patient’s  needs  are  identified,  assigned  a 
value  weighted  integer,  and  transformed  into  nursing  care  hours. 
Physicians’  orders  have  a  direct  impact  on  nursing  care  hours. 

In  the  civilian  sector,  nursing  managers  have  recognized  the 
need  to  monitor  costs  and  identify  potential  areas  for  change,  in 
attempts  to  decrease  expenditures.  Acuity  systems  are  being  used  not 
only  to  determine  staffing,  but  also  to  estimate  direct  nursing  costs 
under  associated  Diagnosis  Related  Groups.  Acuity  systems  can  be 
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used  to  monitor  the  degree  to  which  physician  orders  impact  on 
musing  care  costs. 

The  Army  nursing  community  has  established  a  workload 
management  system  to  perform  some  of  the  same  functions  as  those 
of  the  civilian  patient  classification  and  patient  acuity  systems.  The 
Army  system  has  been  automated  and  is  being  used  primarily  by 
nursing  staff  managers  to  determine  and  project  workload  needs. 
Administrators  outside  of  the  nursing  department  are  becoming 
interested  in  the  system  because  "overall,  this  system  provides  a 
corporate  tool  which  will  allow  the  best  use  of  people,  as  well  as 
estimate  future  staffing  needs  based  on  changes  such  as:  expanding 
services,  changing  missions,  or  an  increase  in  the  intensity  of  care" 
(Vail,  1985,  p.  2). 

History.  The  need  to  match  the  patient’s  needs  to  available 
nursing  staff  has  existed  for  as  long  as  nursing  itself.  Florence 
Nightingale  wrote  that  "one  sick  person  is  often  waited  on  by  four 
with  less  precision,  and  is  really  less  cared  for  than  ten  who  arr 
waited  on  by  one. . ."  (Nightingale,  1859,  p.  24). 

Patient  classification  literature  in  the  United  States  dates  from 
1922,  when  the  New  York  Academy  of  Medicine  recommended  five 
hours  and  four  minutes  of  nursing  care  in  a  twenty-four-hour  period 
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for  each  patient.  In  1936,  the  National  League  of  Nursing  Education 
recommended  dividing  the  amount  of  time  required  per  patient  into 
eight  age  and  disease  categories.  The  first  recorded  time  that 
patients  were  actually  classified  based  on  required  care  was  in  1960, 
at  Johns  Hopkins  University  (Kirby,  1986).  Part  of  the  drive  toward 
classification  systems  came  from  the  results  of  two  studies  on  the 
relationship  between  nurse  staffing  and  patient  welfare.  These  two 
studies,  conducted  about  twenty-five  years  ago,  showed  that  although 
each  patient  received  more  direct  nursing  care  the  average  level  of 
patient  welfare  did  not  rise.  Later  studies,  in  the  1970s,  focused  on 
the  differences  between  primary  nursing  and  team  nursing  (Flood  & 
Diers,  1988).  Beginning  in  the  late  70s  and  continuing  to  this  time, 
articles  about  patient  classification  systems  revolve  around  the 
creation  of  new  systems  or  the  evaluation  and  comparison  of  older 
systems. 

In  1980,  Vaughn  and  MacLeod  divided  the  reasons  for  having  a 
patient  classification  system  into  two  categories  of  objectives.  They 
called  these  "short-term  "  and  "long-term"  objectives.  The  "short-term" 
objective  balances  manpower  requirements  with  staff  for  each  area 
and  shift.  The  "long-term"  objective  summarizes  actual  versus 
required  staff  over  a  period  of  time,  for  the  purpose  of  hiring 
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decisions.  "Most  hospitals  in  the  United  States  currently  use  some 
form  of  patient  classification  system  to  determine  nursing  staffing 
requirements  for  short-range  and/or  long-range  planning. . 

(Hoffman,  Schaefer,  and  Zuraikal,  1986,  p.  13).  The  current  point  of 
contention  is  which  system  is  best  for  whom  and  how  management 
can  use  the  selected  system  to  make  both  "short"  and  long-term" 
decisions  (Vaughn  &  MacLeod,  1980). 

The  increased  use  of  patient  classification  systems  within 
hospitals  has  led  to  substantial  duplication  of  effort.  Numerous 
systems  have  been  developed,  each  of  which  may  be  used 
within  only  one  hospital.  To  wish  to  develop  one’s  own  system 
is  understandable  since  it  is  important  that  the  instrument 
accurately  measure  what  it  is  intended  to  measure:  the  extent 
of  patient  dependency  on  nursing  care  under  different  unit 
conditions.  The  numbers  of  staff  required  to  provide  that 
nursing  care  is  influenced  by  architectural  and  operating 
features  of  the  nursing  unit,  medical  care  practices,  musing 
care  standards  and  policies,  etc.  (Williams,  1988,  p.  90) 

In  the  past,  nursing  care  costs  were  included  in  the  charge  for 
room  and  board.  "Today,  nursing  departments  actively  seek  ways  of 
identifying  nursing  care  costs  so  that  musing  can  be  seen  as  a  valid 
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revenue  center"  (Grohar,  i^yers,  &  McSweeney,  1986,  p.  19).  A 
further  impetus  to  identify  the  real  cost  of  nursing  came  with  the 
implementation  of  the  Diagnosis  Related  Group  (DRG)  prospective 
payment  system,  itself  a  patient  classification  system.  Improved 
efficiency  is  driving  the  need  for  information.  Patient  classification 
systems  are  seen  as  one  method  of  supplying  that  need,  because  they 
provide  an  objective  method  to  assess  patients*  needs.  Furthermore, 
the  nurse  is  seen  as  an  "honest  broker"  for  the  patient,  with  no 
individual  gain  to  be  derived  from  the  system  (Speer,  1990),  making 
nursing  classification  systems  less  suspect  than  some  of  the  other 
classification  systems  available. 

In  1981,  JCAH  Standard  II  for  Special  Care  Units  required  that 
"each  special  care  unit  shall  be  properly  directed  and  staffed  according 
to  the  nature  of  the  special  patient  care  needs  anticipated  and  the 
scope  of  services  offered"  (Ambutas,  1987,  p.  364).  Further,  in  1986 
the  standards  set  forth  by  the  Joint  Commission  on  Accreditation  of 
.'Tospitals  (JCAH)  stated  that  the  nursing  department  will  "define, 
implement,  and  maintain  a  system  for  determining  patient 
requirements  for  nursing  care  on  the  basis  of  demonstrated  patient 
needs,  appropriate  nursing  intervention,  and  priority  of  care"  (JCAH, 
1986,  p.  133).  Even  more  recently,  in  the  1990  Accreditation  Manual 
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for  Hospitals  Standard  NR  4.4,  the  Joint  Commission  for  the 
Accreditation  of  Health  Care  Organizations  (JCAHO)  stated  that  "The 
nursing  department/service  defines,  implements,  and  maintains  a 
system  for  determining  patient  requirements  for  nursing  care  on  the 
basis  of  demonstrated  patient  needs,  appropriate  nursing  intervention, 
and  priority  for  care"  (JCAHO,  1989,  p.  130).  These  standards  seem  to 
mandate,  legitimize,  and  guarantee  the  continuance  of  the  various 
systems  of  patient  classification. 

Types  of  Patient  Classification  Systems.  Usually  patient 
categories  are  based  on  the  amount  of  care  required  (Rum an,  Krueger, 
&  Nelson,  1988).  Acuity  is  most  often  based  on  the  severity  of  illness, 
and  is  a  direct  reflection  of  the  amount  of  nursing  care  required. 
Severity  has  as  many  definitions  as  there  are  writers.  Some  of  the 
most  common  were  collected  by  Thomas  and  Longo  in  their  article 
Application  of  Severity  Measurement  Systems  for  Hospital  Quality 
Management : 
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Varying  Definitions  of  Severity 


Author 

Brewster  et  at  (1985) 

Coffee  and  Goldfarb  (1986) 

Gonella,  Hornbook,  and  Louis  (1984) 

Horn  (1986) 

Knaus  et  al.  (1981). 

Kreitzer,  Liebner,  &  Roveti  (1982) 

Schweiker  (1982) 

Smits,  Fetter,  &  McMahon  (1984) 


Definition  of  Severity 
"potential  for  organ  failure" 

"disease  progression  toward 
death  (or  physical  impairment, 
if  death  is  an  unlikely  outcome 
for  a  specific  disease)" 

"risk  of  death  or  temporary  or 
permanent  impairment" 

“quantification  of  patient's  total 
burden  of  illness" 

"probability  of  mortality” 

"degree  and  impact  of  change 
on  the  health  status  of  an 
individual  as  a  result  of  illness 
or  injury" 

"risk  of  immediate  death  or 
permanent  loss  of  function" 

"impact  of  the  particular 
disease  process  on  patient’s 
physiologic  integrity" 


(Thomas  &  Longo,  1990,  p.  223) 

If  the  differences  in  definitions  were  not  already  confusing 
enough,  the  systems  that  have  grown  as  measures  of  severity  from 
these  definitions  are  even  more  diverse.  A  good  feel  for  the  diversity 
comes  from  a  summary  description  by  Iezzoni  in  her  article  entitled 
Using  Severity  Information  for  Quality  Assessment:  A  Review  of  Three 
Cases  by  Five  Severity  Measures: 
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Summary  Description  of  Severity  Measures 


Severity  Measure 

Data  Requirements 

Definition  of  Severity 

Diagnosis  Used 

Classification  System 

APACHE  II 

Chart  review 

Risk  of  imminent  death 

Mo 

Score  from  0  to  71  indicating 
overall  patient  severity. 

Computerized 
Severity  Index 

Chart  review 

Treatment  difficulty 
presented  to  physicians 

Yes 

Score  from  1  to  4  for  patient 
overall  and  for  each  ICD-9-CH 
diagnostic  code. 

Disease  Staging 
Q-Scale 

Discharge  abstract 

Resource  needs 

MO 

Percentage  compared  with  an 
average  of  100  indicating  overall 
patient  severity. 

Critical  Criteria 
Version 

Chart  review 

Risk  of  death  or 
functional  impairment 

Yes 

Stage  from  1.0  to  4.0  with 
sifcetages  for  each  disease 
present. 

KedisGroups 

Chart  review 

Risk  of  imminent  organ 
failure 

No 

Score  from  0  to  4  on  admission 
and  morbidity  score  at  mid-stay 
review  indicating  overall  severity. 

Patient  Management 
Categories 

Discharge  abstract 

Resource  needs 

Yes 

Cost-basad  relative  weight 
compared  with  an  average  of  1.0; 
also  classifies  based  on  major 
surgery. 

(Iezzoni,  1989,  p.  376) 

"Acuity",  "category",  and  "patient  classification"  mean  different  things 
to  different  people,  yet  they  someximes  mean  exactly  the  same  thing. 
They  all,  however,  have  some  common  elements:  (a)  a  method  for 
grouping  patients,  (b)  guidelines  for  classification,  (c)  patient  care  time 
required  by  that  classification,  and  (d)  some  sort  of  mathematical 
summary. 

The  two  most  common  types  of  classification  systems  are 
prototype  and  factor  evaluation.  The  prototype  system  provides 
broad  descriptions  of  characteristics  for  a  patient  which  are 
typical  for  a  certain  category,  and  the  patient  is  placed  into  the 
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category  which  most  closely  represents  him.  Factor  evaluation 
has  a  number  of  critical  indicators  that  are  separately  rated 
numerically  and  then  combined.  The  total  numerical  value 
places  the  patient  into  a  certain  category.  (Ambutas,  1987,  p. 
364) 

In  1984,  Shroeder,  Rhodes,  and  Shields  conducted  a  study  to 
compare  two  very  different  nursing  acuity  systems.  Their  study 
revealed  little  difference  between  the  systems  in  regard  to  the  staffing 
requirements  identified.  Although  a  fair  amount  of  argument  still 
exists  as  to  the  merits  of  the  various  systems  available,  a  definite 
move  is  being  made  toward  a  common  or  standard  system. 

Major  Classification  Systems.  Five  mayor  patient  classification 
systems  have  emerged  in  recent  years.  The  first  of  these,  DRGs,  are 
perhaps  the  most  recognized. 

Diagnosis  Related  Groups  (DRGs)  were  developed  by  a  group  of 
researchers  at  Yale  University  to  identify  a  set  of  case  types, 
each  representing  a  class  of  patients  with  similar  processes  of 
care  and  receiving  a  predictable  bundle  of  services  (i.e.,  a 
product)  from  an  acute  care  hospital.  (Rosko,  1988,  p.  257) 

To  date,  two  different  DRG  systems  have  been  used  to  effect  the 
prospective  payment  system  in  the  state  and  federal  sectors. 
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In  1980,  the  State  of  New  Jersey  operated  the  first  DRG-based 
hospital  rate  regulation  program.  Under  this  system, 
prospective  payment  rates  were  established  for  383  DRGs 
which  were  based  on  ICDA-8  nomenclature.  In  1983,  the 
Medicare  program  implemented  a  prospective  pricing 
mechanism  based  on  a  second  generation  DRG  system  which 
consists  of  patient  categories  and  uses  the  ICD-9-CM  diagnostic 
codes.  (Rosko,  1988,  p.  258) 

It  is  felt  that  "deficiencies  in  the  ICD-9-CM  coding  system  directly 
affect  the  equity  of  the  Medicare  payment  system"  (McMahon  & 

Smits,  1986,  p.  662).  The  principal  problem  involves  the  inability  of 
the  DRG  system  to  reflect  patient  acuity.  In  1985,  in  a  rather 
successful  attempt  to  resolve  this  problem,  patient  costs  were 
predicted  by  combining  nursing  acuity  data  with  DRG  data.  This 
method  resulted  in  an  89  percent  accuracy  rate  (Curtin,  1985). 

Around  the  same  time,  the  second  patient  classification  £ystem 
was  emerging,  in  the  Department  of  Defense  (DOD)  environment.  A 
study  was  conducted  under  the  auspices  of  the  Health  Care  Studies 
Division,  Academy  of  Health  Sciences,  Fort  Sam  Houston,  Texas.  This 
study,  published  in  September  1981,  was  intended  to  "develop  an 
improved  patient  classification  system  which  would  provide  a  better 
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staffing  mix  based  on  quantified  direct  nursing  care  requirements  for 
critical  care,  medical/surgical,  obstetric,  psychiatric,  neonatal,  and 
pediatric  inpatient  clinical  services"  (Sherrod,  Rauch,  &  Twist,  1981, 
v).  This  system,  now  called  the  Workload  Management  System  for 
Nursing  (WMSN),  was  directed  to  be  used  throughout  the  DOD 
system  on  March  21, 1989.  The  WMSN  has  recently  been  automated. 
The  automated  WMSN  is  called  the  Workload  Management  System 
(WMS).  WMSN  is  being  used  as  a  patient  classification  standard  and 
as  a  manpower  standard.  The  WMSN  combines  an  objective  factor- 
evaluation  patient  classification  system  with  a  staffing  methodology 
(Vail,  Norton,  &  Rimm,  1986). 

Researchers  from  Thomas  Jefferson  Medical  College  and 
Systemetrics,  Inc.  developed  the  third  system,  a  disease  staging 
protocol,  using  levels  of  severity. 

This  system  classifies  diseases  rather  than  health 
problems. . . .  Diseases  are  defined  by  clinicians  in  specific 
detail,  including  location  of  the  problem,  manifestation  of  the 
medical  problem,  cause  of  the  problem,  and  severity  of  the 

problem _ In  general,  the  four  stages  of  disease  (cancer  has 

five)  can  be  described  in  ascending  order  of  severity  as:  Stage 
1)  Conditions  with  no  complications,  or  problems  of  minimal 
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severity;  Stage  2)  Problems  limited  to  an  organ  or  organ  system 
with  increased  probability  of  complications;  Stage  3) 

Generalized  systemic  involvement,  poor  prognosis;  Stage  4) 
Death.  (Rosko,  1983,  p.  259) 

The  fourth  mqjor  patient  classification  system  presently  on  the 
scene  was  developed  by  researchers  at  Johns  Hopkins  University  and 
is  called  the  Severity  of  Illness  Index  (SSI). 

The  SSI  is  an  ordinal  measure  of  patient  severity  consisting  of 
four  levels.  The  assessment  of  severity  depends  upon  the 
scores  tor  the  following  seven  variables: 

1.  Stage  of  the  principal  diagnosis 

2.  Complications  of  the  principal  diagnosis 

3.  Concurrent  interacting  conditions  that  affect  the  hospital 
episode  of  treatment 

4.  Dependency  on  the  hospital  staff 

5.  Extent  of  non-operating  room  procedures 

6.  Rate  of  response  to  therapy  or  rate  of  recovery 

7.  Residual  impairment  remaining  after  therapy  for  the  acute 
aspect  of  the  hospitalization.  (Rosko,  1988,  p.  260) 

Training  for  this  system  starts  with  an  intensive,  three-day  session. 
Follow-up  training  is  required,  and  effort  has  been  exerted  to  get 
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inter-rater  reliability.  An  offshoot  of  this  index  is  the  Computerized 
Severity  Index  (CSI). 

In  the  CSI  system  a  patient  is  placed  in  one  of  four  ordinal 
categories  of  severity  for  each  disease  on  the  basis  of 
comorbidity,  complications,  and  interactive  effects  of  conditions. 
The  CSI  uses  information  based  on  laboratory  values,  vital  signs, 
radiological  findings,  and  other  clinical  information  found  in  the 
medical  record  but  not  summarized  in  current  discharge  abstract 
forms.  Computer  software  for  this  system  was  developed  in 
1986.  (Rosko,  1988,  p.  261) 

Approximately  800  Patient  Management  Categories  (PMCs)  were 
developed  by  researchers  from  Blue  Cross  of  Western  Pennsylvania, 
using  50  disease-specific  panels  of  specialist  and  generalist  physicians. 
PMCs  represent  the  fifth  and  last  mqjor  patient  classification  system. 
The  PMC  protocol  relies  primarily  on  diagnosis  and  secondarily 
upon  clinical  procedures.  Age  and  gender  serve  as  additional 
classification  criteria  for  only  a  few  PMCs.  Recognizing  that 
physicians  treat  symptoms  and  suspected  conditions  that  may 
not  be  suggested  by  the  principal  diagnosis,  the  PMC  system  also 
relies  on  the  admitting  diagnosis  and  secondary  diagnoses  to 
classify  patients.  The  PMC  system  also  recognizes  that  patients 
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with  multiple  diagnoses  can  be  dichotomized  as  those  who  are 
(1)  subject  to  a  single  disease  process,  and  (2)  comorbid.  (Rosko, 
1988,  p.  162) 

Of  the  mqjor  patient  classification  systems  being  used,  only  one 
of  them,  the  DOD  WMSN,  is  used  exclusively  (or  even  mainly)  to 
facilitate  nursing  management.  All  of  the  systems,  however, 
could  be  adapted  to  include  nursing,  or  even  to  target  nursing 
information-gathering  as  a  major  goal. 

Evaluating  Patient  Classification  Systems. 

In  1978,  the  United  States  Public  Health  Service’s  Division  of 
Nursing  published  a  manual  for  evaluating  hospital  staffing 
requirements  based  on  the  results  of  an  eight-year  study  done  at 
San  Juaquin  General  Hospital  Despite  the  growing  number  of 
patient  dassiiication  systems  in  use,  Vaughn  and  MacLeod 
advocated  a  standardized  system  for  hospitals  across  the  country. 
On  the  other  hand,  Giovanetti  believed  that  patient  classification 
systems  should  be  tailored  to  meet  the  unique  needs  of 
individual  health  care  facilities.  (Vail,  Morton  &  Rieder,  1987,  p. 
290) 

Although  there  seems  to  be  a  diversity  of  opinion  about  patient 
classification  systems  that  is  as  varied  as  there  are  systems  in 
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existence,  Rosko  stated  in  1988  that  they  will  all  provide  value  under 
the  following  circumstances: 

1.  The  classification  system  must  be  based  on  information 
that  is  reliable. 

2.  Patients  who  consume  resources  of  relatively  equivalent 
values  should  be  grouped  together. 

3.  The  number  of  categories  should  be  manageable. 

4.  The  classification  system  must  be  clinically  meaningful  to 
physicians. 

5.  Information  used  to  classify  patients  must  be  easily  access 
ible  at  a  reasonable  cost.  (Rosko,  1988,  p.  262) 

Present  Situation.  Because  the  DRG  system  has  become  so 
pervasively  used  in  the  United  States,  it  has  been  evaluated  by  nurses 
to  ascertain  its  reliability  in  capturing  nursing  workload. 

Of  mqjor  concern  to  nursing  leaders  are  questions  relating  to  the 
potential  impact  of  Diagnostic  Related  Groups  (DRGs)  on  the 
nursing  professions,  particularly  with  respect  to  nurse  staffing 
issues.  Questions  such  as  ‘Will  the  implementation  of 
prospective  cost  reimbursement  mechanisms  based  on  DRGs 
render  the  measurement  of  nursing  workload  futile?’  and  ‘Do 
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DRGs  adequately  reflect  nursing  workload?'  are  frequently 
asked.  (Giovarmetti,  1985,  p.  88) 

The  concerns  of  Giovannetti  in  1985  were  reiterated  in  1988  by 
Thompson  and  Diers  when  they  stated  that  "an  inquiry  is  necessaiy  at 
this  time  because  it  is  held  that  the  Diagnosis  Related  Group 
Prospective  Payment  System  fails  to  account  for  variations  in  the 
intensity  of  nursing  services  among  patients  in  different  DRGs" 
(Thompson  &  Diers,  1988,  p.  473).  A  system  that  cannot  capture  a 
major  resource  cost  such  as  nursing  cost  would  need  to  be  integrated 
with  an  "acuity/severity"  system  that  could  severity-adjust  and  thus 
capture  differences  in  cost  efficiency. 

Some  method  to  meld  a  severity  system  with  DRGs  has  been 
sought  since  the  advent  of  diagnosis-based  and  resource-based  patient 
classification  systems. 

A  complicating  factor  in  the  study  of  the  relationship  between 
DRGs  and  nursing  workload  measures  is  that  the  two 
classification  schemes  do  not  employ  the  same  unit  of  analysis. 
As  previously  noted,  DRGs  are  determined  on  a  per  case  basis, 
whereas  nursing  workload  measures  yield  a  coefficient  per  shift 
or  day  based  ou  an  aggregate  of  cases.  It  would  seem  that 
nursing  workload  measures  need  to  be  reduced  to  a  case-specific 
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level  in  order  to  compete  with  the  precedent  set  by  the  DRGs. 

It  is  not  difficult  to  obtain  this  type  of  information,  but  few 
hospitals  have  policies  or  procedures  to  retain  this  data  as  part 
of  the  patient’s  permanent  record.  (Giovannetti,  1985,  p.  90) 
Presently,  patient  classification  systems  are  being  used  to  justify 
manpower,  improve  personnel  utilization,  distribute  workload,  and 
monitor  nursing-related  issues  such  as  documentation  and  quality 
assurance  (Norton  &  Jones,  1990).  Other  uses  of  patient  classification 
systems  include  establishing  historical  trend  analysis  of  acuity  levels, 
defining  diagnosis-related  work  factors,  combining  diagnosis  work 
factors  and  acuity  factors,  setting  staffing  classification  ratios 
according  to  acuity  analysis,  predicting  inpatient  pharmacy  workload, 
establishing  quality  assurance  programs  and  comparing  physicians’ 
practice  patterns  for  efficiency.  Lundgren  and  Daniels  (1986)  write 
that  "the  index  of  patient  acuity  of  illness  and  the  number  of  standard 
hours  of  nursing  care  are  good  predictors  of  pharmacy  workload  of  the 
same  and  following  days;  the  potential  exists  to  use  these  nursing 
workload  indicators  in  determining  pharmacy  staffing  requirements" 
(p.  2453-2459). 

An  even  more  extensive  and  probably  the  newest  use  for  patient 
classification  systems  is  to  monitor  physician  performance.  "By 
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gathering  severity  data,  hospital  administrators  who  have  struggled 
with  sagging  bottom  lines  now  have  firm  statistics  to  challenge  their 
medical  staffs  on  the  need  to  control  costs'*  (Brightbill,  1990,  p.  26). 
One  company,  lameter,  Inc.,  is  using  its  own  system,  the  Acuity  Index 
Method  (AIM),  to  help  a  hospital  and  its  medical  staff  analyze  clinical 
quality.  As  one  author  writes,  "The  system  reports  demonstrate  how 
these  adjusted  data  can  be  used  initially  to  document  plhysician  and 
hospital  quality  and  cost  efficiency,  as  well  as  monitor  their  respective 
performance  concurrently"  (Mohlenbrock,  1989,  p.  4). 

In  the  past,  one  of  the  drawbacks  for  patient  classification 
systems  was  their  complicated,  time-consuming  nature.  With  the 
advent  of  the  desktop  computer,  all  stages  of  the  categorization 
process,  from  collecting  the  raw  data  to  reporting  results,  can  be  and 
are  being  automated. 

Purpose  of  the  Study 

The  purpose  of  this  study  is  to  solve  the  problem  of  the  lack  of 
physician  knowledge  about  the  patient  classification  system  used  by 
DOD,  the  WMSN.  The  objectives  of  the  study  were  to:  determine 
what  the  physician  needs  to  know,  and  create  a  method  for  teaching 
it;  and  evaluate  the  change  in  physician  knowledge  that  is  expected  to 
result  from  the  education. 
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CHAPTER  II 

Method  and  Procedures 

The  first  step  of  the  study  was  to  determine  what  sets  of  WMSN 
knowledge  were  required  by  physicians.  The  sets  of  WMSN 
knowledge  required  by  physicians  are  represented  by  the  questions  on 
the  survey  instrument  (APPENDIX  B).  The  sets  of  knowledge  and 
even  specific  questions  were  gleaned  from  interviews  with  nurses 
familiar  with  the  WMSN  (Graham,  Sylvester,  Brazil  &  Worden)  at 
Brooke  Army  Medical  Center  (BAMC),  Fort  Sam  Houston,  Texas, 
through  observation  at  both  BAMC  and  Irwin  Army  Community 
Hospital,  Fort  Riley,  Kansas,  and  through  literature  research.  The 
survey  questions  were  validated  by  WMSN  Project  nurse  (Worden) 
and  Quality  Assurance  nurse  (Brazil)  at  BAMC. 

The  second  step  of  the  study  was  to  generate  an  initial 
curriculum  for  physicians  based  on  the  knowledge  sets  in  the  survey 
instrument.  This  material  was  taught  to  a  group  of  physicians.  The 
physicians  took  a  pre-test  and  a  post-test,  to  assist  in  validating  the 
change  in  their  knowledge  and  to  adjust  the  material  to  their  needs. 
Two  iterations  of  curriculum  development  occurred.  A  final 
curriculum  was  developed  based  on  lessons  learned  during  the 
teaching  phase.  Descriptive  statistics  were  used  to  assist  in  the 
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evaluation  of  the  pre-  and  post-test  changes  after  the  initial 
curriculum  had  been  taught.  Systat ,  a  statistical  package  for  personal 
computers  by  Systat,  Inc.,  was  used  to  process  the  data.  The 
curriculum  and  the  survey  instrument  were  revised  based  on  the  pre- 
and  post-test  changes  and  interviews  with  the  physicians  after 
completion  of  the  teaching  phase.  The  revised  survey  is  at 
Appendix  D. 
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CHAPTER  III 
Results 

This  study  involved  the  creation  of  a  curriculum  for  physicians, 
complete  with  WMSN  pre/post  test,  handouts  and  visual  aids.  The 
revised  curriculum  is  at  Appendix  C.  The  pre/post  test  was  designed 
to  provide  indicators  for  change  or  adjustment  of  the  curriculum  based 
on  the  knowledge  of  the  students.  The  class  for  physicians  was 
taught  on  a  Friday  afternoon  and  lasted  two  hours.  The  pre-test  and 
identical  post-test  were  given  at  the  beginning  of  the  two-hour  session 
and  at  the  end,  respectively.  The  pre-  and  post-tests  consisted  of  the 
questions  found  in  Appendix  B,  the  survey  instrument.  The  pre-  and 
post-tests  were  called  pre-  and  post-survey  instruments.  One  person 
graded  both  the  pre-  and  post-surveys.  The  surveys  were  graded  in 
the  same  way,  regardless  of  pre-  or  post-  placement.  The  surveys 
were  graded  with  a  liberal  interpretation  of  what  was  right.  Specific 
words  or  answers  were  not  counted  as  the  only  possible  "right" 
answers.  In  ideal  situations,  the  pre-survey  instrument  should  be 
given  at  least  two  weeks  prior  to  the  class,  both  to  prevent  pre-test 
bias  to  learning  and  to  evaluate  the  pre-knowledge  of  the  physicians 
involved  and  adjust  the  curriculum  to  the  physicians’  specific  needs. 
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Descriptive  statistics  (means,  standard  deviations,  range, 
skewness,  and  kurtosis)  for  the  pre-test,  post-test,  and  both  tests 
combined,  are  presented  in  Appendix  E.  A  more  complete  report  of 
pre-  and  post-test  descriptive  statistics,  including  the  mean  of  the 
correct  and  incorrect  answers  for  both  tests,  as  well  as  the  range, 
mean,  and  mode  for  both  tests,  are  presented  at  Appendix  F.  Based 
on  these  statistics,  none  of  the  distributions  appeared  to  depart 
markedly  from  a  normal  distribution.  Univariate  frequency 
distributions  were  examined  for  each  test  to  identify  possible  outliers. 
No  adjustment  was  necessary  to  correct  for  outliers. 

To  investigate  the  number  of  people  getting  correct  and 
incorrect  answers  to  each  question,  the  frequency  distributions  at 
Appendixes  G  and  H  were  assembled.  Further  frequency  distributions 
by  score  (numbers  of  correct  answers)  for  each  test  are  placed  at 
Appendixes  I  and  J.  All  of  the  frequency  distributors  were  used  to 
create  a  frequency  distribution  that  indicates  the  number  of  people 
who  changed  from  an  incorrect  answer  on  the  pre-test  to  a  correct 
answer  on  the  post-test,  by  question  number.  All  of  the  descriptive 
statistics  were  then  used  to  evaluate  the  curriculum  and  the  survey 
instrument  and  make  changes  as  indicated  by  the  results. 
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CHAPTER  IV 
Discussion 

Based  on  the  descriptive  statistics  presented  in  the  Results 
section  and  discussions  with  physicians  and  nurses  at  Irwin  Army 
Community  Hospital,  the  curriculum  and  the  survey  instrument  were 
revised.  The  change  in  means  from  the  pre-test  to  the  post-test 
indicated  that  as  a  whole  the  physicians  at  this  facility  did  not  initially 
know  veiy  much  about  WMSN,  but  that,  after  being  taught,  their 
knowledge  base  changed.  No  test  was  used  to  see  if  the  change  from 
a  mean  of  3.83  on  the  pre-test  to  a  mean  of  10.91  on  the  post-test  was 
significant  in  a  statistical  sense,  because  the  only  real  test  of 
significance  of  change  in  this  case  is  if  the  physicians’  practice 
patterns  change  to  use  nursing  resources  more  effectively,  and  that  is 
outside  the  scope  of  this  project.  However,  during  post-instruction 
discussions  with  nurses  in  this  facility,  there  seem  to  be  some 
indicators  of  physician  interest  change.  Some  muses  report  that  their 
physicians  have  been  asking  questions  about  the  system  and  wanted 
to  see  the  WMSN  reports  on  patients  and  on  the  ward.  This  had 
never  happened  before,  and  may  or  may  not  be  the  result  of  the 
instruction  on  WMSN,  but  it  is  nonetheless  seen  as  a  healthy 
communication  trend. 
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A  mean  change  that  was  also  investigated  was  the  change  in 
means  for  correct  and  incorrect  answers.  The  means  of  correct  and 
incorrect  answers  almost  totally  reversed  themselves  from  the  pre¬ 
test  to  the  post-test.  This  was  further  construed  to  establish  that  the 
curriculum,  although  needing  some  revision,  was  basically  sound  and 
did  not  need  mqjor  revision. 

The  survey  instrument  was  changed  based  mainly  on  discussions 
with  test  takers  after  completion  of  the  instruction  process  and  the 
post-test.  These  discussions  revealed  that  some  of  the  questions  did 
not  convey  to  physicians  what  was  intended,  so  the  questions  were 
revised  and  rediscussed  with  all  physicians  surveyed.  Final 
adjustments  were  then  made  to  the  survey  instrument. 

The  frequency  distribution  system  of  displaying  correct  and 
incorrect  answers  to  the  survey  instrument  for  both  tests  was  used  to 
indicate  areas  to  be  changed.  Of  particular  use  was  the  distribution  of 
change  in  answers  from  the  pre-test  to  the  post-test.  The  survey 
questions  that  showed  the  least  change  from  an  incorrect  answer  on 
the  pre-test  to  a  correct  answer  on  the  post-test  were  questions  six 
and  seven.  Further  investigation  using  the  frequency  distribution  by 
test  question  shows  that  question  number  six  changed  veiy  little  from 
pre-  to  post-test,  because  15  of  the  23  people  surveyed  got  a  correct 
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answer  on  the  pre-test.  The  curriculum  was  adjusted  downward  on 
emphasis  about  why  we  need  a  nursing  workload  management 
system,  because  the  physicians  here  already  seem  to  have  a  grasp  of 
this  information.  Investigation  into  the  lack  of  change  in  question  14 
revealed  the  most  serious  deficiency  in  the  curriculum.  The  pre-test 
indicated  that  91  percent  of  the  people  did  not  know  the  information 
required  for  a  correct  answer,  and  furthermore,  that  even  after 
getting  instruction,  74  percent  still  did  not  get  the  correct  answer. 

The  concept  of  WMSN  figures  being  an  average  was  not  sufficiently 
emphasized  in  the  curriculum.  Question  14-even  with  changes  to  the 
curriculum  to  show  this  concept  more  directly  (by  filling  out  the 
actual  acuity  forms),  and  a  direct  statement  about  averages  being  the 
base  of  the  system-will  probably  still  be  a  discriminator  of 
understanding.  The  curriculum  and  the  survey  instrument  were  also 
changed  based  on  receipt  of  over  50  percent  correct  answers  on  the 
pre-test  and  over  50  percent  incorrect  answers  on  the  post-test. 

A  follow-on  study  to  indicate  the  need  for  this  kind  of  program 
at  other  DOD  facilities  may  reveal  the  need  for  either  individual 
curriculums  at  each  facility  based  on  the  knowledge  of  the  physicians, 
or  a  central  program  taught  at  a  site  such  as  the  Academy  of  Health 
Sciences,  Fort  Sam  Houston.  An  even  further  follow-on  cost  benefit 
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analysis  of  a  centralized-versus-decentralized  system  may  be  indicated 
if  the  first  study  reveals  a  DOD-wide  need  for  a  WMSN  curriculum  for 
physicians. 


Educating  Physicians 

33 


CHAPTER V 

Conclusions  and  Recommendations 

This  project  has  resulted  in  the  creation  of  a  curriculum  for 
physicians  about  the  Workload  Management  System  for  Nurses.  A 
survey  instrument  has  been  created  as  a  built-in  evaluation  system  to 
assist  in  changing  the  curriculum  as  physician  needs  change. 

Based  on  the  results  of  the  study  and  discussions  with  physicians 
and  nurses  at  Irwin  Army  Community  Hospital  about  the  study,  it  is 
the  recommendation  of  this  researcher  that  the  curriculum  be  taught 
to  new  or  incoming  physicians  who  do  not  score  at  least  75  percent  on 
the  survey  instrument,  and  to  all  others  who  wish  to  learn  more 
about  the  system.  Two  hours  appears  to  be  enough  time  to  teach  the 
material,  and  the  material  (as  presently  revised)  seems  to  meet  the 
present  skill  level  of  the  physicians  at  Irwin  Army  Community 
Hospital. 

A  further  recommendation  is  that  all  medical  facilities  using 
WMSN  evaluate  the  degree  of  knowledge  of  their  physicians  about  the 
WMSN.  It  is  clear  that  very  few  physicians  at  Irwin  Army 
Community  Hospital  know  anything  at  all  about  the  system  that 
determines  nursing  resources  and  that  is  impacted  heavily  by  their 
practice  patterns.  If  this  situation  is  similar  at  other  military  medical 
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facilities,  then  the  curriculum  presented  by  this  project  needs  to  be 
revised  based  on  physician  needs  within  the  facility  and  taught 
accordingly. 

If  nursing  resources  are  to  be  used  in  the  most  effective  manner, 
physicians  must  understand  and  appreciate  the  importance  of  their 
practice  patterns  on  cost  in  terms  of  nursing  workload  requirements. 
Training/education  for  physicians  regarding  their  impact  on  nursing 
workload  must  be  either  provided  from  some  outside  source  or  taught 
at  the  facility.  If  effective  exchange  of  information  regarding  practice 
patterns  between  nurses  and  physicians  does  not  occur  at  each  facility 
as  a  part  of  the  natural  communication  process,  then  it  must  be 
institutionalized  as  part  of  the  education/training  system  at  each 
facility  or  as  a  part  of  a  central  curriculum  at  the  Academy  of  Health 


Sciences. 
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Appendix  A 
DEFINITIONS 

1.  Workload  Management  System  for  Nurses  (WMSN)  (pronounced 
"Wims"): 

A  patient  classification  instrument  which  requires  a  registered 
nurse  to  assess  ten  groups  of  factors  related  to  direct  patient  care 
and  assign  an  overall  score  to  each  factor.  This  assessment  is  used 
to  predict  care  requirements  for  a  24-hour  period.  The  scores  are 
weighted  and  summed  to  classify  each  patient  into  one  of  seven 
categories  (0-6).  Indirect  care  time  is  already  calculated  into  the 
methodology.  The  WMSN  staffing  methodology  is  used  to 
determine  nursing  care  requirements  as  well  as  the  mix  of 
personnel  required  for  quality  care. 

2.  Classification : 

A  means  of  grouping,  ordering,  or  arranging  objects  or  concepts 
into  sets,  based  on  the  relationships  among  the  objects  or  concepts. 

3.  Inpatient  classification: 

Refers  to  the  different  methods  and  procedures  used  to  group 
patients  according  to  their  requirements. 
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4.  Acuity  system : 

A  tool  that  provides  data  based  on  the  severity  of  illness  of  the 
patient. 

5.  Prototype  classification  system : 

Broad  descriptions  of  characteristics  for  a  patient,  to  determine 
their  category. 

6.  Factor  evaluation  classification  system : 

A  process  in  which  separately  rated  critical  indicators  are 
numerically  combined  to  place  the  patient  in  a  category. 

7.  Direct  nursing  care  time : 

Time  required  for  activities  taking  place  in  the  presence  of  the 
patient  and/or  the  patient’s  family. 

8.  Indirect  nursing  care  time: 

Time  required  for  activities  performed  away  from  the  patient 
and/or  the  patient’s  family. 

9.  Nursing  care  hour  requirement: 

Number  of  hours  of  nursing  time  required  to  provide  quality  care. 
This  is  determined  using  precalculated  patient  care  hour 
requirement  charts  in  the  following  clinical  areas: 
Medical/Surgical,  Pediatric,  Critical  Care,  Psychiatric,  OB/GYN, 
and  Nursery. 
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Appendix  B 

SURVEY  INSTRUMENT 

1.  What  role  do  physicians  play  in  the  Workload  Management 
System  for  Nursing  (WMSN)? 

2.  Do  your  orders  raise  and/or  lower  patient  "acuity  levels"  as 
defined  by  the  WMSN? 

3.  According  to  WMSN,  which  category  of  patient  requires  a  one- 
staff-to-one-patient  ratio? 

4.  Describe  one  way  the  WMSN  affects  the  quality  of  health  care. 

5.  Is  the  WMSN  used  at  Irwin  Army  Community  Hospital? 

6.  Name  one  reason  why  we  need  a  nursing  workload  management 
system. 

7.  Name  one  way  we  know  that  the  WMSN  is  reliable,  valid  and 
quantifiable. 
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8.  Is  the  WMSN  research-based? 

9.  Name  one  agency  that  requires  the  WMSN. 

10.  What  categories  of  patients  would  necessitate  coordination  with 
the  ward  nurse  before  placement  in  the  ward? 

11.  Name  a  WMSN  measurable  aspect  of  care  that  is  ordered  entirely 
by  the  physician. 

12.  Name  a  WMSN  measurable  aspect  of  care  that  is  ordered  entirely 
by  the  nurse. 

13.  What  information  is  provided  by  ward  Nursing  Care  Hours? 

14.  Is  the  staffing  recommended  by  WMSN  inflated,  optimal, 
maximum,  average,  or  minimum  needed? 

15.  Are  WMSN  standards  developed  for  all  areas  in  the  hospital 
nursing  environment? 
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Answer  Key  to  Survey  Instrument 

1.  Physicians’  orders  drive  6  of  the  10  factors  in  evaluating  patient 
"acuity."  Physicians  admit/transfer/and  discharge  patients  taking 
into  account  nursing  care  hour  requirements. 

2.  Yes,  they  both  raise  and  lower  patient  "acuity  levels." 

3.  Category  5. 

4.  Justifies  increased  staffing  needed  to  provide  the  right  numbers 
and  mix  of  staff  to  care  for  patient’s  requirements. 

5.  Yes,  all  DOD  hospitals  use  WMSN. 

6.  JCAHO  requirements  and  staffing  requirements. 

7.  Inter-rater  reliability  is  done  regularly. 


8.  Yes. 
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9.  JCAHO  and  DOD. 

10.  All  patients,  especially  Categories  4-6. 

11.  Vital  signs,  monitoring,  IV  therapy,  treatments/procedures/medi¬ 
cations,  and  respiratory  therapy. 

12.  Teaching,  emotional  support,  and  most  of  activities  of  daily  living. 

13.  Hours  of  nursing  care  required  and  the  staffing  needed  to 
accomplish  the  requirements. 

14.  Average. 

15.  No,  standards  are  not  available  for  the  emergency  room,  labor  and 
delivery,  recovery,  and  outpatient  clinics. 
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APPENDIC  C 
Revised  Curriculum 
With  Overheads  and  Handouts 

The 

WORKLOAD  MANAGEMENT  SYSTEM  FOR  NURSING 

(WMSN)  or  (WMS) 

Pronounced  "WIMS" 
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Learning  Objectives 

1.  Feel  an  urgency  to  understand  my  impact  on  WMSN  and  WMSN’s 
impact  on  me  as  a  physician. 

2.  Be  able  to  describe  WMSN  briefly  to  someone  else. 

3.  Know  that  WMSN  is  mandated  by  the  provisions  of  JCAHO. 

4.  Have  some  familiarity  with  the  creation  of  the  WMSN  system. 

5.  Incorporate  this  new  knowledge  about  WMSN  into  my  practice 
pattern  as  a  physician. 
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Teaching  Objectives 

1.  Prepare  convincing  arguments  concerning  why  physicians  need  to 
know  about  WMSN. 

2.  Provide  a  short  description  of  WMSN. 

3.  Tell  who  requires  the  system. 

4.  Outline  the  history  of  WMSN. 

5.  Show  how  WMSN  works,  using  the  data  sheets. 

6.  Tell  what  WMSN  is  used  for  and  what  it  could  be  used  for. 

Agenda 

(A  two-part  class  to  physicians,  consisting  of  three  hours,  with  a  break 
in  between) 

1.  Why  do  you  need  to  know  about  WMSN? 

2.  What  is  WMSN? 

3.  What  mandates  the  system? 

4.  What  is  the  history  of  WMSN? 

5.  How  does  WMSN  work? 

6.  What  is  WMSN  used  for? 


7.  What  could  WMSN  be  used  for? 


Overhead 

[1] 

[2] 

[3a] 

[3b] 
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Teaching  Script 


Tide  Slide 

A.  What  are  we  going  to  talk  about,  and  how  will  the  program  hc> 
broken  up? 


B.  Why  do  physicians  need  to  know  about  the  WMSN? 


1.  Some  people  don’t  want  you  to  know,  for  fear  that  you  will 
manipulate  the  system  to  get  what  you  want.  Others  do  want 
you  to  know  and  want  to  foster  partnership,  not  warfare. 
(Example:  Some  insurance  agencies  use  medisgroups,  and 
the  information  they  derive  they  will  not  share  with  the 
doctors.) 

2.  Physicians’  orders  drive  over  half  of  the  ten  factors  in 
evaluating  patient  acuity  levels. 

3.  It  is  a  reliable,  prospective  indicator  to  physicians  of  when 
nursing  does  not  have  the  resources  to  provide  the  amount 
and/or  kinds  of  nursing  care  required.  Also,  it  is  an  indicator 
of  cost  efficiency  or  inefficiency  of  the  physician’s  practice 
pattern. 

4.  It  is  the  staffing  standard  used  by  the  U.S.  Army  Manpower 
Authorizations  and  Documentation  Agency  (USAMARDA) 
now  integrated  with  the  U.S.  Army  Force  Integration  Support 
Agency  (USAAFISA). 

5.  When  there  are  not  enough  nurses,  physicians’  workload 
increases. 

6.  When  there  are  not  enough  nursing  personnel,  interesting 
cases  are  lost  through  CHAMPUS. 

7.  Quality  of  care  diminishes  when  nursing  personnel  are  not 
available.  Physicians  get  blamed  for  lack  of  quality  care,  even 
though  they  are  an  episodic  presence  in  the  life  of  the  patient, 
with  the  musing  staff  providing  the  continuity  of  care.  A 
study  indicates  that  "infection  rates  occurred  more  often  on 
the  short-staffed  unit"  (Flood  &  Diers,  1988,  p.  42). 
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8.  Residency  programs  are  threatened  if  there  are  not  adequate 
numbers  of  nursing  support  staff. 

9.  Patient  classification  systems  that  do  not  make  clinical  sense 
to  the  physician  do  not  provide  value  to  the  institution 
(Rosko,  1988,  p.  262).  The  value  of  WMSN  is  lessened  if 
physicians  don't  buy  into  it 

10.  In  some  military  hospitals,  admissions  are  limited  based  on 
WMSN  statistics. 


C.  What  is  WMSN? 

[4]  It  is  a  two-part  inpatient  acuity  classification  system  that  first 
assesses  patients  using  10  groups  of  factors  and,  based  on  that 
assessment  assigns  patients  to  one  of  7  categories. 

[5]  Secondly,  WMSN  links  the  acuities  with  a  staffing  methodology 
for  required  nursing  staff. 

The  patient  classification  system  that  physicians  are  the  most 
aware  of  is  the  Diagnosis  Related  Group  system  (DRG).  DRGs 
are  based  on  diagnosis  alone  and  do  not  take  into  account  patient 
acuity.  WMSN,  on  the  other  hand,  does  not  involve  diagnosis  at 
all,  but  is  based  on  what  Horn  in  1986  called  the  "quantification 
of  a  patient’s  total  burden  of  illness." 

Presently,  nursing  researchers  are  attempting  to  combine  the 
DRG  and  WMSN  systems. 


D.  What  mandates  the  system? 

The  Joint  Commission  on  Accreditation  of  Health  Care 
Organizations  (JCAHO)  mandates  that:  "The  nursing 
department/service  defines,  implements,  and  maintains  a  system 
for  determining  patient  requirements  for  musing  care  on  the 
basis  of  demonstrated  patient  needs,  appropriate  nursing 
intervention,  and  priority  for  care."  It  further  mandates  that 
"the  hospital  admissions  system  must  allow  for  participation  from 
the  nursing  department  and/or  service  in  coordinating  patient 
requirements  for  nursing  care  with  available  nursing  resources." 

There  seems  to  be  a  basic  need  for  some  system  to  describe 
severity/acuity.  We  have  problems  defining  severity. 


[7a-e] 
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[8a-d]  Many  people  have  tried  to  describe  or  define  severity  of  illness. 

The  first  group  of  overheads  show  five  different  severity  systems 
now  being  used.  The  second  set  of  overheads  shows  some  of 
these  systems  in  more  detail. 

[9a-b] 


[10] 


[11] 


c.  Workload  Management  &  System  for  Nursing  (WMSN), 
Vail,  Norton,  and  Rimm,  1984. 

The  nursing  care  hours  study  involved  37,000  time 
measurements  to  establish  mean  times  for  direct  nursing  care 
activities.  This  study  was  later  validated  by  the  consulting  firm 
Health  Management  Systems  Associates. 

The  second  study  used  objective  criteria  to  measure  the 
percentage  of  indirect  care  time  and  the  amount  of  unavailable 
time. 

In  January  1982,  the  chief  of  the  Army  Nurse  Corps  directed  the 
development  of  a  single  system  of  patient  classification. 

From  1982  through  1985,  the  Army/Navy  muse  researchers 
joined  forces  to  develop  the  WMSN  to  meet  their  patient 
classification  needs. 


E.  History  of  WMSN. 

1.  In  the  past,  nurse  staffing  was  based  on  occupied  bed  days. 

2.  In  the  1950s,  the  U.S.  Army  instituted  a  four-category  patient 
classification  system.  Staffing  was  still  based  on  occupied  bed 
days,  but  some  attention  was  given  to  the  four  categories: 
Intensive  Nursing  Care  (Cat  I),  Moderate  Nursing  Care  (Cat 
II),  Minimal  Nursing  Care  (Cat  HI),  and  Self  Care  (Cat  IV). 

3.  WMSN  is  based  on  three  studies  conducted  from  1981-1984. 

a.  U.S.  Army  Nursing  Care  Hours  Standards  Study ,  Sherrod, 
Rauch,  and  Twist,  1981. 

b.  Indirect  Nursing  Care  Time,  Misener,  Frelin,  and  Twist, 
1983. 
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[12] 


4.  In  1985.  the  WMSN  was  tested  at  five  sites  and  implemented. 
The  primary  function  of  this  system  was  to  provide 
management  information  to  all  levels  of  nursing  management 
for  effective  allocation  and  utilization  of  nursing  personnel. 

5.  In  December  1986.  WMSN  was  incorporated  into  an  Army 
manpower  staffing  standard,  MS-1IL 

6.  In  September  1987.  WMSN  was  accented  as  a  Joint 
Healthcare  Manpower  Standard  for  inpatient  nursing  units. 

7.  In  1988.  the  automation  of  WMSN  was  completed,  using  the 
Uniformed  Chart  of  Accounts  Personnel  Utilization  System 
(UCAPERS)  and  Zenith  PCs. 

F.  How  does  WMSN  work? 


(Hand  out  patient  acuity  data  sheets.) 

[13-14] 

1.  The  process  begins  with  classification  of  patients  by  an 

ANC/RN  into  categories,  using  the  patient  acuity  worksheets. 
In  the  general  sheet  there  are  10  factors  and  in  the 
psychiatric  worksheet  there  are  9  factors. 


(Show  overheads.) 

[14a-o]  (Explain  how  sheets  are  filled  out) 

A  factor  is  simply  an  item  on  the  patient  acuity  sheet  that 
reflects  a  need  for  nursing  care.  Sometimes  people  call  some 
of  the  factors  critical  indicators.  A  critical  indicator  is  a 
factor  that  has  the  greatest  impact  on  direct  care  time. 


[15-16] 


(Hand  out  patient  acuity  sheets  15  &  16. 

Show  slides  16a-l  Explain  how  sheets  differ  from  the  general  ones.) 


[16a-i] 

[17] 


The  data  used  to  fill  out  the  acuity  sheets  comes  from: 

a)  Physician  orders/medical  care  plans 

b)  Nurses  orders/nursing  care  plans 
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c)  Hospital/unit  standards  of  care  or  SOPs 

d)  Prediction  by  the  RN  of  care  required  over  a 
24-hour  period 


Just  looking  at  the  factors  on  the  acuity  sheet  tells  any  physician 
that  he/she  has  a  nuyor  impact  At  least  six  of  the  eight  critical 
indicators  are  driven  by  the  physician’s  ordering  plan.  In 
addition  to  having  a  mqjor  role  in  the  final  classification  of  the 
patients,  the  physician  also  admits,  transfers,  and  discharges  the 
patient  hopefully  taking  into  account  the  nursing  care  hour 
requirements  associated  with  each  patient  A  physician’s  orders 
can  directly  affect  the  raising  or  lowering  of  the  patient’s  acuity 
levels. 


I  have  seen  physicians  who,  in  an  attempt  to  lower  acuity  levels, 
"DC  emotional  support”  This  does  not  work,  because  teaching 
and  emotional  support  are  two  of  the  factors  that  are  driven  not 
by  physicians,  but  by  nurses,  as  is  most  of  the  factor  "activities  of 
daily  living."  Physicians  control  most  of  the  other  factors- vital 
signs;  monitoring;  IV  therapy;  treatments,  procedures, 
medications,  and  respiratory  therapy.  Each  point  on  the  acuity 
sheet  is  equal  to  7.5  minutes  of  direct  nursing  care  time. 


Direct  care  time  is  defined  as  the  "time  required  for  activities 
taking  place  in  the  presence  of  the  patient  and/or  family." 
Indirect  nursing  care  time  is  built  into  the  methodology  and  can 
be  considerable.  Indirect  nursing  care  time  is  defined  as  "time 
required  for  activities  and  tasks  performed  away  from  the  patient 
and/or  family.'’ 

[19]  (The  percentages  show  how  much  indirect  car?  time  is  needed  in  the  different  specialty  areas.) 

[20]  It  includes  such  things  as  charting,  phone  coordination,  cleaning, 
admissions,  care  conferences,  personal  time  and  staff  education. 
In  addition  to  the  other  factors  involved,  290  hours  or  full  time 
administration  is  allotted  to  head  nurses  and  wardmasters  in 
CONUS. 


[21] 

[22] 
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[23]  There  are  some  special  considerations  on  points  given  to 
inpatient  wards  that  care  for  outpatients  or  do  after-hours 
recovery  or  one-day  surgeries.  Once  the  total  points  for  a  patient 
are  added,  you  look  to  section  II  of  the  acuity  sheet  to  get  the 
categoiy  of  the  patient 

[24] 

There  are  seven  categories.  Category  O  is  a  patient  on  pass  who 
is  gone  the  entire  24  hours.  You  can  see  that  once  a  patient 
reaches  a  Categoiy  IV,  coordination  with  the  musing  staff 
becomes  even  more  necessary.  A  Categoiy  V  patient  requires 
one-to-one  muse/patient  time,  and  Category  VI  patients  need 
continued  care  plus  another  staff  member  as  well. 

[25] 

Once  you  know  the  categories  of  all  your  patients  in  the  specific 
areas,  you  accumulate  the  categories  and  determine  according  to 
specialty  in  the  seven  clinical  areas  the  nursing  care  hour 
requirement. 

[26] 

The  nursing  care  hours  are  then  added  and  the  personnel 
requirements  are  gotten  from  the  daily  personnel  requirements 
chart. 

[27] 

Nursing  care  hour  requirements  tell  you  how  many  hours  on  the 
average  of  nursing  care  are  required,  and  how  many  staff  of  what 
mix  are  needed.  The  staffing  recommended  in  this  method  is  an 
average. 

[28]  In  order  to  keep  the  validity  of  the  WMSN  high,  Interrater 
Reliability  Testing  is  done.  An  IRR  of  85%  is  the  present 
accepted  standard  of  adequacy. 

[23]  2.  Standards  for  workload  do  not  presently  exist  for  the  ER, 

L&D,  post-anesthesia,  and  outpatient  care. 

[30]  3.  The  entire  system  was  worked  over  by  both  inhouse  and 

outhouse  statisticians  and  found  to  be  highly  valid  and 
reliable.  The  system  is  obviously  quantifiable. 

[31,  32,  33,  34,  35] 

4.  Here  are  some  of  the  WMSN  reports  that  are  used  in  various 
military  hospitals. 
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[36] 

G.  The  automated  WMSN  ia  used  at  Irwin  Army  Community 
Hospital  It  is  used  at  all  DOD  hospitals  and  in  some  civilian 
facilities.  Its  present  uses  include  distribution  of  workload, 
improving  personnel  utilization,  and  assessing  nursing  quality 
assurance. 


H.  What  is  the.future  of  WMSN? 

[37] 

The  last  potential  use  is  probably  the  most  interesting  to 
physicians.  Various  civilian  agencies  across  the  nation,  including 
industry,  insurance  companies  and  hospitals,  are  now  using  some 
sort  of  severity  index  to  severity-adjust  for  case  mix  and  in  an 
attempt  to  change  physicians’  clinical  practice  patterns. 

[38-39-40] 

Data  is  available  to  payors  of  health  care,  such  as  the  data  on  the 
handouts.  This  data  is  being  used  to  examine  and  compare  cost 
Dhvsidans. 

[41-42-43-44] 

Third-party  payors  are  seeking  what  they  are  calling  the  most 
efficient  provider. 

[46] 

This  provider  is  defined  as  the  provider  that  can  give  maximum 
quality  at  maximum  cost  efficiency.  Payors  are  maintaining 

[46]  that  cost  control  is  70  to  80  percent  under  the  control  of  the 
physician.  They  assert  that  resources  are  largely  allocated  by  the 
physician,  and  that  there  are  large  variations  in  cost,  depending 
on  the  physician’s  practice  patterns.  Claims  are  made  that  the 

[47]  largest  tendency  to  overuse  resources  occurs  the  most  often  at 
the  lower  acuity  levels,  rather  than  the  higher  levels. 

(Source  lecture  by  Dr.  Mohlenbrock  (Iameter,  Inc.) 


L  Conclusion. 

The  Workload  Management  System  for  Nursing  (WMSN)  is 
largely  driven  by  your  ordering  pen.  You  affect  the  use  of 
resources  and  the  provision  of  those  resources.  If  the  military 
ever  decides  to  use  the  WMSN  to  severity  adjust  your  cases-as 
the  civilian  world  is  beginning  to  do- then  the  WMSN  could 
potentially  affect  you  as  much  as  you  affect  it. 


THE  WORKLOAD  MANAGEMENT 

SYSTEM  FOR 
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COMPUTERIZED  SEVERITY  INDEX 

Developed  by  Susan  D.  Horn  of  Q.C.  Inc. 


Sourco:  Haalth  Weak,  March  26,  1990 


COMPUTERIZED  SEVERITY  INDEX 

Source:  HealthWeek,  March  26,  1990 
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PNEUMONIA 


COMPUTERIZED  SEVERITY  INDEX 

Source:  Health  Week,  March  26,  1990 
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WORKLOAD  MANAGEMENT  SYSTEM  FOR  NURSING 
Interrater  Reliability  (IRR)  Testing 


1.  Hospital 


5.  Sample  Size  Selection  Table 


4.  IRR  Rater 


<  5 

>.  6  and  <.  20 
_>  21  and  <.  30 
>.  31  and  <.  40 
>  41 


Sample  Size 

all  patients 
6 

7 

8 
9 


SEE  INSTRUCTIONS 
ON  REVERSE  BEFORE 
COMPLETING. 


6.  Category  Agreement 

f  (#  Agreed)  x  100%  f  _  x  10< 

e(#  Sampled)  e 

Provide  the  MEPRS  Office  with  this  IRR  Category  Agreement  score  and  the  date. 


a.  Patients 


x  100%  = 


%  Agreement 


b.  Original 
Category 


c.  Irr 

Category 


DA  FORM  5391-R, 


EXPANSION  OF  THE  WORKLOAD 
MANAGEMENT  SYSTEM  FOR  NURSING 
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POST  ANESTHESIA 
OUTPATIENT  CARE 


HIGHLY  VALID 

RELIABLE  AND  QUANTIFIABLE 
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AVERAGE  DAILY  COMPARISON  OF  REQUIRED  STAFF  TO  ACTUAL  STAFF  BY  WORK  CENTER 
Facility:  FT  TOMCAT  Section:  ALL  SECTIONS 

For  the  aonth  of  March  1587 


‘.Average 

Actual  Daily  Staff! 

Total 

Required 

« 

Parent 

»  _ 
e 

Avg  Daily 

Avg  Dally  Dlff- 

Diff- 

Section 

Staff  iRegular  Overtime 

Borrowed: 

■Staff 

Staff 

•rtnet 

trance 

CRIT  CARE 

• 

rn  : 

27.3 

0.0 

e 

0.0 

27.3 

30.0 

-8.7 

-31X 

LPN  : 

19.5 

0.0 

0.0 

19.5 

14.7 

4.8 

24X 

na  : 
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0.0 
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9.9 

-4.9 

-98Z 
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lpn  : 

22.8 
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40.4 

-32.3 
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42.3 

23.5 

18.8 

OU 

na  : 

13.1 

0.0 

0.0 

13.1 

50.3 

-37.2 

-398X 

FACILITY  TOTAL  i 

117.2 

0.0 

0.0 

1  117.2 

130.9 

-19.7 

-10X 

•  Difference  «  Total  Daily  Avg  -  Required  Daily  Avg 

Date/Tine:  10  Apr  87  tS:90:01  Page:  9 


Sample  Report  by  Facility 


PREPARED:  DAILY  SUMMARY  REPORT 


PATIENTS  BY  CATEGORY 
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NURSING  WORKLOAD  MANAGEMENT  SYSTEM 
HOSPITAL  PROFILE 
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USE  OF  WMSN 

INFORMATION 

JUSTIFY  CURRENT  MANPOWER 

JUSTIFY  ADDITIONAL 
PERSONNEL 

IMPROVE  UTILIZATION 
OF  PERSONNEL 

DISTRIBUTE  WORKLOAD 

MONITOR  SPECIFIC  NURSING 
RELATED  ISSUES: 

DOCUMENTATION 
NURSING  QA 


POTENTIAL  USES 
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COMPARE  PHYSICIANS  HEALTH  CARE  PRACTICES  FOR  EFFICIENCY 


INTEGRATION  OF  HOSPITAL  SERVICE  QUALITY 
WITH  OBJECTIVE  CLINICAL  QUALITY 
AND  COST  EFFICIENCY  MEASURES 
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5.  Establish  a  plan  for  ongoing  monitoring  of  clinical 
quality  and  cost  efficiency  which  includes  feedback 
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HOMETOWN  GENERAL  HOSPITAL 
1257  Genesee  Avenue,  Southwest 
Hometown,  California  92037 


UNIFORM  HOSPITAL  DISCHARGE  DATA  SET 


Patient  Name  Louis  Underwood 

Date  of  Birth  04/13/21 

Address  39., Madison 

street 

Home  Phone  16191475-3025 
Occupation 

Medical  Insurance  Co.  Medicare 
Referring  Physician 
Transferring  Facility 
Attending  Physician 
Service 

Admitting  Diagnosis 
Principal  Diagnosis 
Secondary  Diagnoses 


Age  ££  '  Sex  M 

Hometown  £A  22021 

city  state  zip 

Work  Phone  _ 

Employed  By  _ 

Group  Number  . 


M,.  Anderson 
CcntnLYaIley.HPiPiial 
C^oodheqn 
Carding  Sornery 
Chest  gain 

Subendocardial  Myocardial  Infarction 
COED 

Aortig  Stenosis 


Principal  Procedure  Double  Internal  Mammary  Coronary  Artery  Bypass 
Secondary  Procedures  Cardiac  Catheterization 
Evimpnarv.Aniipram 


Date  of  Admission  02/20/86  Date  of  Discharge  02/26/86 

Discharge  Status  (Home,  Transfer,  Dead)  £  Post  Mortem?  K 

Total  Charges  S45.689 
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Date  of  Admission  03/07/86  Date  of  Discharge  03/21/86 

Discharge  Status  (Home,  Transfer,  Dead)  Home  Post  Mortem? _ 

Total  Charges  HL23.1 


VOLUME  AND  MORTALITY  INDEX 

Internal  Medicine 


LOW  Mortality  Index  HIGH 


CHARGE  AND  ALOS  INDEX 

Internal  Medicine 


ALOS  INDEX  HIGH 


COMPARATIVE  PHYSICIAN  PROFILES 

DEG  121:M.I.  W/COMPLICATIONS 
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J 


PHYSICIAN  NUMBER 


COMPARATIVE  PHYSICIAN  PROFILES 

DRG  121:M.I.  W/COMPLICATIONS 


PHYSICIAN  NUMBER 


Relationship  Between  Clinical  Quality 
And  Patient  Resource  Consumption 
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Resource  Consumption  (Charges) 


TOTAL  INPATIENT  CHARGES 
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APPENDIX  D 

Revised  Survey  Instrument 


1.  Do  physicians  have  a  large  or  a  small  influence  on  the  Workload 
Management  System  for  Nursing? 

2.  Can  orders  written  by  physicians  after  the  patient  acuity  levels 
have  been  calculated  contribute  to  lower  nurse  staffing? 

3.  According  to  WMSN,  which  category  of  patient  requires  a  one- 
staff-to-one-patient  ratio? 

4.  Describe  one  way  the  WMSN  affects  the  quality  of  health  care. 

5.  Is  the  WMSN  used  at  your  hospital? 

6.  Name  one  reason  why  we  need  a  nursing  workload  management 
system. 

7.  How  docs  the  WMSN  check  for  accuracy  of  patient  category 
coding? 

8.  Is  WMSN  research-based? 

9.  What  specific  categories  of  patients  would  especially  require 
coordination  with  the  ward  nurse  before  placement? 

10.  Name  a  WMSN  measurable  aspect  of  care  that  is  ordered  entirely 
by  the  nurse. 

11.  Is  the  staffing  recommended  by  WMSN  inflated,  optimal, 
maximum,  average,  or  minimum  needed? 

12.  Are  WMSN  standards  developed  for  all  areas  in  the  hospital 
nursing  environment? 
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Revised  Survey  Instrument  Key 


1.  Large. 

2.  Yes. 

3.  Category  5. 

4.  Justifies  increased  staffing  (numbers  and  mk)  to  provide  quality 
care. 

5.  Yes,  all  DOD  hospitals  use  WMSN. 

6.  JCAHO  requirements  and  staffing  requirements. 

7.  Inter-rater  reliability  testing  is  performed  regularly,  using  an 
independent  assessment  person. 

8.  Yes. 

9.  Categories  4-6. 

10.  Teaching,  emotional  support. 

11.  Average. 
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APPENDIX  E 


Means,  Standard  Deviations,  Skewness 
and  Kurtosis  Comparisons 


Measure 

N 

Mean 

SD 

Skewness 

Kurtosis 

Pre-Test 

23 

3.83 

3.03 

0.885 

0.496 

Post-Test 

23 

10.91 

3.25 

-0.687 

-0.597 

Both  Tests  Combined 

46 

7.37 

4.74 

0.073 

-1.333 
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APPENDIX  F 

Descriptive  Statistics  for  Pre-  and  Post-Test 


Pre-Test  Post-Test 


N  of  Cases 

= 

23 

23 

Minimum 

= 

0 

4 

Maximum 

= 

12 

15 

Range 

= 

12 

11 

Median 

= 

2 

12 

Mean 

s 

3.826 

10.913 

Mode 

= 

2 

12,13 

Variance 

= 

9.150 

10.538 

Standard  Deviation 

= 

3.025 

3.246 

Standard  Error 

= 

0.631 

0.677 

Skewness 

= 

0.885 

0.687 

Kurtosis 

= 

0.496 

0.598 

Co-Variance 

= 

0.791 

0.297 

Mean  of  Correct  Answers 

= 

5.933 

16.733 

Mean  of  Incorrect  Answers 

= 

17.133 

6.266 
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APPENDIX  G 

Frequency  Distribution  by  Test  Question 
for  Pre-Test 


Question 


No.  of  No.  of 
People  People 
Getting  Getting 
Correct  Incorrect 
Answer  Answer 


Percent  Percent 
Correct  Incorrect 


1 

5 

18 

21.74 

78.26 

2 

12 

11 

52.17 

47.83 

3 

2 

21 

8.70 

91.30 

4 

10 

13 

43.48 

56.52 

5 

4 

19 

17.39 

82.61 

6 

15 

8 

65.22 

34.78 

7 

1 

22 

4.35 

95.65 

8 

13 

10 

56.52 

43.48 

9 

8 

15 

34.78 

65.22 

10 

6 

17 

26.09 

73.91 

11 

3 

20 

13.04 

86.96 

12 

3 

20 

13.04 

86.96 

13 

2 

21 

8.70 

91.30 

14 

2 

21 

8.70 

91.30 

15 

2 

21 

8.70 

91.30 
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APPENDIX  H 

Frequency  Distribution  by  Test  Question 
for  Post-Test 


Question 

No.  of 

1"» _ »  _ 

rcupie 

Getting 

Correct 

Answer 

No.  of 

Feopie 

Getting 

Incorrect 

Answer 

Percent 

Correct 

Percent 

Incorrect 

1 

16 

7 

69.57 

30.43 

2 

21 

2 

91.30 

8.70 

3 

11 

12 

47.83 

52.17 

4 

18 

5 

78.26 

21.74 

5 

23 

0 

100 

0 

5 

16 

7 

69.5', 

30.43 

7 

16 

7 

69.57 

30.43 

8 

23 

0 

100 

0 

9 

17 

6 

73.91 

26.09 

10 

18 

5 

78.26 

21.74 

11 

18 

5 

78.26 

21.74 

12 

19 

4 

82.61 

17.39 

13 

11 

12 

47.83 

52.17 

14 

6 

17 

26.09 

73.91 

15 

18 

5 

78.26 

21.74 

Educating  Physicians 

143 


APPENDIX  I 

Frequency  Distribution  by  Score* 
for  Pre-Test 


Number  of 
Correct  Answers 


Number  of  People  Percent  of 

Receiving  That  No.  People  Getting 

of  Correct  Answers  Correct  Answers 
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APPENDIX  J 

Frequency  Distribution  by  Score* 
for  Post-Test 


Number  of 
Correct  Answers 


Number  of  People 
Receiving  That  No. 
of  Correct  Answers 


Percent  of 
People  Getting 
Correct  Answers 


4 

1 

4.35 

5 

1 

4.35 

6 

1 

4.35 

7 

2 

8.70 

9 

2 

8.70 

10 

1 

4.35 

11 

2 

8.70 

12 

4 

17.39 

13 

5 

21.74 

14 

1 

4.35 

15 

3 

13.04 

N=23 


*  Score  =  Number  of  correct  answers 
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APPENDIX  K 

Frequency  Distribution  of  Change  in  Answers 
From  Pre-  to  Post-Test 


Numbers  of  People 
Who  Changed  From 
An  Incorrect  Answer 
on  the  Pre-Test  to  a 
Correct  Answer  on 
the  Post-Test 
- 

4 

8 

9 

9 

9 

9 

10 

11 

12 

15 

15 

16 
16 
19 


Question  Number 
6 

14 
4 

13 

9 

3 

2 

8 

1 

10 

11 

7 

15 
12 

5 


